Skip to content

feat(providers): add opt-in local inference provider adapters - #48

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-4from
feat/local-inference
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-4from
feat/local-inference

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in local inference providers: ollama, vllm, lmstudio, llamacpp and sglang. Each is an ordinary OpenAI-compatible upstream, active only when its base URL variable is set. With all five unset (the default), the provider list, catalog, model routes and every response are unchanged.

Provider Base URL Optional key
ollama OLLAMA_BASE_URL OLLAMA_API_KEY
vllm VLLM_BASE_URL VLLM_API_KEY
lmstudio LM_STUDIO_BASE_URL LM_STUDIO_API_KEY
llamacpp LLAMA_CPP_BASE_URL LLAMA_CPP_API_KEY
sglang SGLANG_BASE_URL SGLANG_API_KEY
  • Addressing: models are addressed as <provider>:<model> on the existing managed /v1/chat/completions, /v1/responses and /v1/models routes. Only the OpenAI-compatible /v1 API is used; Ollama's native /api/chat is not.
  • URL validation:
    • A bare origin gets /v1.
    • Embedded credentials, queries, fragments, traversal and paths outside /v1 are rejected.
    • Plain HTTP is accepted only for loopback, RFC 1918, IPv6 unique-local and localhost addresses.
    • Public HTTP, wildcard, multicast and link-local metadata addresses are rejected.
    • A malformed URL disables that provider and logs the variable name once.
  • Keys: a provider key comes from the private provider credential loader. If no key is configured, an explicitly configured local server is called with no Authorization header. Gateway authorization is still required. Other providers still refuse when their key is missing.
  • Catalog:
    • Refresh queries only configured servers at /v1/models.
    • Capabilities are declared with provider_catalog:<provider> provenance and never inferred from model names.
    • Catalog entries saved for a disabled provider are hidden.
  • What it doesn't do: no port scanning, discovery, model downloads or inference on the Worker. Behind the Worker, requests run through Flask in the Container, so "loopback" means the Container.

Details are in docs/local-inference.md.

Coordinator edits in this PR

  • routes/provider_credentials.py:_provider_token consults AuthService.provider_requires_api_key before refusing a missing key. This lets an explicitly configured local server run keyless, while keyed providers keep refusing. It is the only change outside the builder's ownership row, and both cases are tested.
  • worker/container-env.mjs: forwards the five base URL and five key variables to the Container.

services/auth_service.py is still 1,230 lines; this PR only adds small call sites there. Moving provider credential loading out of account authentication would be a reasonable follow-up.

Tests

  • New tests in tests/test_local_inference.py cover:
    • the five explicit registrations and their absence when unset;
    • /v1 normalization and URL policy;
    • optional and keyless auth through the real Chat and Responses routes;
    • catalog provenance;
    • no implicit detection.
  • Full Python suite: 3,949 passed, 2 skipped, 1 failed. ruff, mypy and bandit are clean.
    • The failure is an existing flaky test, test_ledger_rows_hold_request_metadata_without_prompts_or_keys. It asserts that the ledger row doesn't contain the string hi, and this run's random key prefix (mllm_yhihlRKN) happened to contain it. The file passes on rerun (11 passed), and this PR doesn't touch it.
  • npm run test:worker: 1,460 passed. npm run test:integration: 7 passed.
  • test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • Live local servers were not exercised. The tests use fake transports.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 5. It is based on integrate/gateway-waves-1-4, the Wave 4 integration branch. The README line comes with the Wave 5 integration PR.

Register ollama, vllm, lmstudio, llamacpp and sglang as OpenAI-compatible providers only when their base URL variable is set (all unset by default). URLs are validated to an explicit /v1 endpoint, plain HTTP is limited to loopback and private addresses, and an optional provider key is used when configured; an explicitly configured local server may be called without one. No port scanning, discovery or downloads; catalog entries carry declared capabilities with provenance.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 44009552-5869-410e-b677-470d5833e681

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy b636129 Oct 09 2026, 01:26 AM

@ALikesToCode
ALikesToCode merged commit 3e28b16 into integrate/gateway-waves-1-4 Oct 9, 2026
4 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the feat/local-inference branch October 26, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant